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REMOTE CONTROL OF A 
MEDICAL DEVICE USING VOICE 
RECOGNITION AND FOOT 
CONTROLS 

Background of Invention 

[0001] The present invention generally relates to a system and method for controlling 
a medical device. More particularly, the present invention relates to a system and 
method for controlling a medical imaging system using a voice recognition and 
foot- controlled system. 

[0002] Medical imaging systems are being used for a growing number of applications 
in the field of medicine. Medical imaging systems are typically used for diagnosis 
as well as for monitoring purposes during surgery. One type of medical imaging 
system commonly used during surgery is an ultrasound imaging system. Typical 
ultrasound systems operate by transmitting ultrasonic sound waves into a patient's 
body using a transducer. The transducer is typically a device placed on the 
patient's body over the area to be imaged that is capable of sending and receiving 
ultrasonic sound waves. The ultrasonic sound waves sent by the transducer are 
reflected by the patient's internal bodily structures. The reflected ultrasonic sound 
waves transmitted into the patient's body are then received by the transducer and 
processed to display a visual representation of the patient's internal bodily 
structures to the surgeon. 

[0003] 

The received ultrasonic sound waves are typically processed by an ultrasonic 
imaging system and displayed in real time on a console for viewing by the 
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examining physician. Typically, the console includes a control console as well as a 
viewing screen on which ultrasonic images are displayed. The control console 
typically includes a number of control devices. The control devices are typically 
manual controls such as dials, switches, knobs, or joysticks, for example, that may 
be used to manipulate the ultrasonic image displayed on the viewing screen. For 
example, the control devices may be used to control the resolution, magnification, 
viewing area, or orientation of the ultrasonic image. In typical ultrasound imaging 
systems, the ultrasonic images may only be adjusted or manipulated solely by the 
control devices. 

[0004] Typically, the control console itself may be fairly large and may take up a 

substantial amount of space in an operating room. However, during surgery, the 
space around the surgeon may often be very limited. For example, multiple trays 
containing operating instruments, as well as other medical devices and support 
personnel may need to be near the surgeon to assist the surgeon during surgery. 
Therefore, there typically is not free space near the surgeon for the ultrasound 
imaging system and console during surgery. Consequently, the ultrasound imaging 
system and console may out of necessity need to be positioned up to several feet 
away from the surgeon and out of the immediate reach of the surgeon. 

[0005] Typically when the console is positioned out of reach of the surgeon, the 
surgeon may be unable to use the control devices to manipulate the ultrasonic 
images during surgery. Because the surgeon is unable to manipulate the ultrasonic 
images during surgery, the surgery may become complicated by limiting the 
adjustability of the ultrasonic image. If the surgeon must adjust the ultrasonic 
image, the surgeon may have to continually physically walk over to the console and 
then adjust the properties of the image using the control devices. Requiring the 
surgeon to continually walk over to the control devices to adjust the ultrasonic 
image may be time consuming, inefficient, or impractical if the surgery requires 
that the surgeon continually monitor the patient. Thus, typical ultrasonic imaging 
systems operable solely by control devices may be undesirable or inefficient for 
use by surgeons in situations where the available space around the surgeon is 
limited. 
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[0006] 



Even if the ultrasound imaging system and console are able to be positioned 



within reach of the surgeon, a number of drawbacks still exist in typical systems 
operable solely by console controls. For example, the surgeon's hands may be in 
use or too busy during the surgery to manually operate the control devices. Thus, 
while the surgeon may be able to reach the control devices, in practicality, the 
surgeon is still unable to use them. Even if the surgeon is able to operate the 
control devices during surgery, a number of drawbacks exist. For example, during 
surgery, the surgeon typically wears latex surgical gloves. The surgeon's gloves 
may often become covered with blood or other materials during surgery. Handling 
the control devices with soiled gloves may contaminate the control devices with 
unitarily materials, which may not present optimal sterile conditions. Thus, the 
control devices may have to be sterilized before, during, and after each surgery 
since the surgeon typically touches the control devices frequently during operation. 
Therefore, the control devices typically must to be made of materials that may be 
handled in a sterile environment. 

[0007] Additionally, even sterilized control devices may be difficult to operate by a 

surgeon. After, the surgeon's gloves become covered with blood or other materials 
during surgery, the gloves may become slippery. Therefore, operation of the 
control devices by hand may become difficult after the surgeon's gloves become 
soiled. Thus, adjusting the control devices by hand in a sterile surgical 
environment may not be the most practical and efficient method of adjusting an 
ultrasonic image. 

[0008] method used to reduce some of the drawbacks associated with the limited 

amount of free space around a surgeon during surgery is the use of a remote 
control system. A remote control system is typically a compact hand-held unit 
including controls such as switches, dials, or joysticks, for example. Remote 
controls may be operable by either one or two hands. The remote control system 
may communicate with the medical imaging system by either a wireless 
transmission system or by a wire-based transmission system. The compact remote 
control system may be used to operate the functionality typically operated by the 
control devices and typically does not require that the control devices be within 
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reach of the surgeon. Thus, the compact size of the remote control takes up less 
free space than the console controls. 

[0009] However, remote control systems may also suffer from some significant 

drawbacks. For example, while the space required for the remote control system is 
typically less than the space required for the entire medical imaging system or 
console controls, some space is still required. Thus, in situations where there is 
very little or no free space around the surgeon even remote control systems may 
still not be a viable alternative. Furthermore, remote controls systems still may 
present the same drawbacks that console controls exhibit with regard to 
sterilization and ease of use as discussed above. Additionally, the controls on the 
remote control system may actually be smaller than the control devices on the 

k Q console to save space and thus the remote control system may be more difficult for 

JrJ a surgeon to manipulate, especially with soiled gloves. 

iy [0010] One method used to reduce some of the drawbacks associated with medical 

imaging systems operable by control devices or remote control systems is the use 
of a foot-controlled console. Foot-controlled consoles are typically comprised of at 
\T\ least one foot-input device such as pedals, switches, or joystick-type devices for 

|* example, that may be operated by the surgeon's foot. In the medical imaging 

H system field, such foot-controlled consoles may be used to control the functions 

5 588 typically controlled by the control devices of the medical imaging system. 

[0011] 

Typically, the foot-controlled console may be placed on the floor near the 
surgeon's feet where operation room space is typically available. Thus, the surgeon 
may still control the functionality of the control console of the medical imaging 
system, via the foot-controlled console, even though the control console may be 
positioned out of reach of the surgeon. That is, although the foot-controlled 
console may not occupy any space in the limited operating space directly around 
the surgeon's hands and torso, the foot-controlled console allows for hands-free 
operation of the functionality of the medical imaging system's control devices by 
the surgeon during surgery. Hands-free operation may reduce the problems 
associated with manually operating control devices or remote controls as discussed 
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above. 

[001 2] However, the typical foot-controlled console system may still suffer from some 
significant drawbacks. One drawback that may occur in typical foot-controlled 
consoles is reduced functionality. That is, the foot-controlled console may lack 
much of the functionality that the control console has. The reduced functionality of 
foot-controlled consoles may occur for a number of reasons. One reason a foot- 
controlled console may lack some of the functionality that the control console has 
is that there is limited space available on a foot-controlled console. Typical control 
consoles may include a large number of control devices for a wide variety of 
features of the medical imaging system. Therefore, in order to accommodate 
control of each feature of the medical imaging system, a large number of foot- 
input devices may need to be placed on the foot-controlled console. Because the 
surgeon's feet are typically larger and less agile than the surgeon's hands, the 
foot-input devices on the foot-controlled console typically may be larger and 
spaced further apart than the corresponding control devices on a typical control 
console. Thus, if the number of control devices on the medical imaging system 
console is high, the corresponding foot-controlled console may become too large 
and inefficient to use in practice. Therefore, in order to keep the foot-controlled 
console compact enough for efficient use, a limited number of foot-input devices, 
typically less than the number of control devices, may have to be placed on the 
foot-controlled console limiting the functionality of the foot-controlled console. 

[0013] |n ap p|j cat j ons ou tside of the medical imaging field, control and functionality 
of various systems have been provided by voice-controlled systems. For example, 
the use of voice-controlled systems has been adopted in fields such as computer 
science to facilitate hands-free operation of personal computers. Typical voice- 
controlled systems utilize a microphone and a voice recognition system. Typically, 
an operator speaks a verbal command into the microphone, and the command is 
then transmitted to the voice recognition system. The voice recognition system is 
typically pre-programmed to recognize the command. After recognizing the verbal 
command, the voice recognition system typically sends a signal to the device being 
controlled to perform the operator's command. Thus, voice recognition systems 
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allow an operator to control a device in a completely hands-free manner. However, 
voice-controlled systems may suffer from some significant drawbacks. These 
drawbacks may make the use of typical voice-controlled systems in the medical 
imaging field very difficult. 

[0014] For example, one drawback that may be present in voice recognition systems is 
the inability to make fine adjustments to continuous controls such as a joystick, 
trackball, or dial, for example. That is, verbal commands typically are not able to 
provide small continuous movements of controls, which may often be required of 
medical imaging systems. Therefore, the use of voice recognition systems may not 
be optimal for surgical applications because of their inability to perform fine 
adjustment. Another drawback is that typical voice recognition systems may react 
slowly to commands. Therefore, in a real-time environment such as during 
:JJ surgery, a significant lag time between a command and execution of the command 

M may complicate the surgery or hinder progress. Furthermore, voice recognition 

systems are typically sensitive to external noise. That is, voice recognition systems 
typically need to be operated in a quiet environment to function properly. Noises 
(□ other than the operator's voice, such as other voices or noise from other devices in 

!^ surgery for example, may cause the voice recognition system to register false 

SU commands. 

;:== [001 5] Thus, a need exists for a medical imaging control system that may be 

efficiently used in the limited operating area of a surgeon. A need further exists for 
a medical imaging control system that may be efficiently used in the limited 
operating area of a surgeon while maintaining all of the functionality available to 
console controls. Additionally, a need exists for a medical imaging control system 
that allows for the efficient control of continuous controls such as joysticks, 
trackballs, or dials, for example, during surgery. 

Summary of Invention 

[0016] 

The preferred embodiment of the present invention provides a system and 
method for controlling a medical imaging system using a voice recognition and 
foot-controlled system. A preferred embodiment of the present invention includes 
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a microphone for receiving and transmitting verbal commands from a surgeon to a 
system control and voice recognition processor. The present invention also 
preferably includes a foot-input console connected to the system control and voice 
recognition processor. To operate the medical imaging system using the voice 
recognition and foot-controlled system, the surgeon preferably speaks a first 
verbal command into the microphone. The first verbal command selects a specific 
function available on the medical imaging system. The surgeon then preferably 
speaks a second verbal command assigning a foot-input device located on the 
foot-input console control over the functionality specified in the first verbal 
command. Once the function specified in the first verbal command is successfully 
assigned to the foot-input device specified in the second verbal command, the 
surgeon may then control the function of the medical imaging device in a hand- 
free manner via the foot during surgery. 

Brief Description of Drawings 

[001 7] Figure 1 illustrates a block diagram of a voice activated, foot-controlled 
medical imaging control system according to a preferred embodiment of the 
present invention. 

[001 8] Figure 2 illustrates a flow chart of the voice activated, foot-controlled medical 
imaging control system according to a preferred embodiment of the present 
invention. 

[001 9] Figure 3 illustrates a block diagram of an alternative embodiment of a voice 
activated, foot-controlled medical imaging control system. 

Detailed Description 

[0020] Figure 1 illustrates a block diagram of a voice activated, foot-controlled 

medical imaging control system 1 00 according to a preferred embodiment of the 
present invention. The medical imaging control system 100 includes a microphone 
1 1 0, a foot input console 1 20, a control console 1 30, a system control and voice 
recognition processor 140, and a medical imaging acquisition and display system 
150. 
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[0021] 



The microphone 1 1 0 is connected to the system control and voice recognition 




processor 1 40. The foot input console 1 20 is also connected to the system control 
and voice recognition processor 140. The control console 1 30 is also connected to 
the system control and voice recognition processor 140. The system control and 
voice recognition processor 140 is connected to the medical imaging acquisition 
and display system 1 50. 

[0022] In operation, a surgeon's voice commands are sent to the system control and 

voice recognition processor 140 via the microphone 1 10 to assign the functionality 
of the control console 1 30 to the foot-input console 1 20. Preferably, the 
microphone 1 10 of the medical imaging control system 100 is attached to a 
headset worn by the surgeon during surgery. The microphone 1 1 0 may be 
connected to the system control and voice recognition processor 140 by either a 
wireless transmission system or by a wire-based transmission system. The foot- 
input console 1 20 of the medical imaging control system 100 is preferably located 
on the floor of the operating room within reach of the surgeon's foot. The foot- 
input console 1 20 preferably includes at least one foot-input device such as a 
pedal or switch, for example. The foot-input console 1 20 may be connected to the 
system control and voice recognition processor 1 40 by either a wireless 
transmission system or by a wire-based transmission system. The control console 
1 30 is typically located in the operating room but out of reach of the surgeon 
because of the limited space around the surgeon. The control console 1 30 
preferably includes at least one control device such as a knob or dial, for example. 
The control devices on the control console 1 30 are typically used by the surgeon to 
adjust the properties of an ultrasound image or perform an imaging function. For 
example, the control devices may be used to print or freeze the image, change the 
focal zone of the image, adjust the contrast or resolution of the image, or adjust 
the orientation of the image. The medical imaging acquisition and display system 
1 50 typically includes a system for generating and displaying medical images for 
the surgeon. The medical images are typically displayed on a LCD screen or 
monitor located in the operating room. 



[0023] 



Although the medical imaging control system 100 is described below with 
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reference to ultrasonic images, the present invention may be used in conjunction 
with any type of medical images such as MRI or GT scan images for example. The 
medical imaging control system 100 described below may be adapted with minimal 
modifications for use with any type of medical imaging by methods know to those 
skilled in the art. 

[0024] In operation of the preferred embodiment where the medical imaging control 
system 100 is used, the surgeon activates the medical imaging control system 100 
through the use of two verbal commands. The first verbal command selects the 
specific functionality of the control console 1 30 to be assigned to the foot- 
controlled console 1 20. The second verbal command then selects the specific foot- 
input device on the foot-controlled console 1 20 to control the selected 
functionality. After giving the two verbal commands, the surgeon may then control 
the functionality of the console control 1 30 with the foot-controlled console 1 20. 

[0025] example, the medical imaging acquisition and display system 1 50 

generates and displays medical images of the patient to the surgeon. During the 
surgery, the surgeon may wish to adjust the properties of an image to get a 
different perspective of the patient's internal structures or to zoom in on a point of 
interest for example. However, as discussed above with regard to the background 
section, because of space limitations or because the surgeon may be using both 
hands, the console control 1 30 of the medical imaging system may be out of reach 
or impractical to use. Thus, in order to adjust the ultrasound images hands-free 
during surgery using the medical imaging control system 100 the surgeon speaks 
a first verbal command into the microphone 1 1 0. The first verbal command spoken 
by the surgeon into the microphone 1 10 preferably is the name of a specific 
function performed by the medical imaging system such as "zoom" for example. 
The microphone 1 1 0 then transmits the signal of the first verbal command to the 
system control and voice recognition processor 140. The system control and voice 
recognition processor 1 40 then compares the signal of the first verbal command to 
a number of pre-programmed signals stored in a database look-up table in the 
system control and voice recognition processor 140. Some signals stored in the 
database look-up table correspond to a specific function of the medical imaging 
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system. If the signal of the first verbal command matches one of the stored 
signals, the functionality spoken in the first verbal command is selected. 

[0026] The surgeon then speaks a second verbal command into the microphone 1 10. 
The second verbal command spoken by the surgeon into the microphone 1 1 0 is 
the name of a specific foot-input device on the foot input console 1 20 such as 
"pedal one" for example. The microphone 1 1 0 then transmits the signal of the 
second verbal command to the system control and voice recognition processor 
140. The system control and voice recognition processor 140 compares the signal 
of the second verbal command to a number of pre-programmed signals stored in 
the database look-up table. Some signals stored in the system control and voice 
recognition processor 140 correspond to specific foot-input devices on the foot- 
controlled console. If the signal of the second verbal command matches one of the 
stored signals, the system control and voice recognition processor 140 assigns the 
foot-input device spoken in the second verbal command to the functionality 
spoken in the first verbal command. Once the system control and voice recognition 
processor 140 assigns the control device functionality to the foot-input device, the 
surgeon may use the foot-input device to adjust the specified property of the 
ultrasound image. 

[0027] Additionally, during the surgery, the surgeon may either reassign the foot- 
input device to the functionality of another control device, or assign the 
functionality of multiple control devices to multiple foot input devices by repeating 
the procedure described above. While the present invention has been discussed 
with reference to a preferred embodiment of controlling a medical imaging system, 
the present invention may be used to control any medical device in a hands-free 
manner. 

[0028] 

Figure 2 illustrates a flow chart 200 of the use of the medical imaging control 
system of Figure 1 according to a preferred embodiment of the present invention. 
First, at step 210, the surgeon selects the functionality of a control device by 
speaking a first verbal command into a microphone 1 1 0. The microphone 1 1 0 then 
transmits the signal of the first verbal command to the system control and voice 
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recognition processor 140. Then, at step 220, the system control and voice 
recognition processor 140 processes the first verbal command by comparing the 
signal of the first verbal command to a number of pre-programmed signals stored 
in the system control and voice recognition processor 140. If the signal of the first 
verbal command matches one of the stored signals, the functionality spoken in the 
first verbal command is selected. Next, at step 230, the surgeon selects the foot- 
input device to control the control device selected at step 210, by speaking a 
second verbal command into the microphone 1 10. The microphone 1 10 then 
transmits the signal of the second verbal command to the system control and voice 
recognition processor 140. At step 240, the system control and voice recognition 
processor 140 then assigns the foot-input device selected at step 230 to the 
control device selected at step 210 by comparing the signal of the second verbal 
command to a number of pre-programmed signals stored in the system control 
and voice recognition processor 140. If the signal of the second verbal command 
matches one of the stored signals, the system control and voice recognition 
processor 140 assigns the foot-input device spoken in the second verbal command 
to the functionality spoken in the first verbal command. Finally, at step 250, once 
the control device functionality is successfully assigned to the foot-input device, 
the surgeon may control the functionality selected at step 21 0 with the foot-input 
device selected at step 230. 

[0029] Figure 3 illustrates a block diagram of an alternative embodiment 300 of the 
medical imaging control system 100 of Figure 1. The alternative embodiment 300 
includes a microphone 310, afoot input console 320, control console 330, a 
system control and voice recognition processor 340, and a medical imaging 
acquisition and display system 350 similarly to Figure 1 above. Additionally, the 
alternative embodiment 300 includes an enhanced display 360. 

[0030] As sjm j| ar |y d escr jbed above with regard to Figure 1 , the microphone 310, the 
foot-input console 320, and the control console 330 are all connected to the 
system control and voice recognition processor 340. The system control and voice 
recognition processor 340 is connected to the medical imaging acquisition and 
display system 350. The enhanced display 360 is connected to medical imaging 
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acquisition and display system 350. 

[0031] In the alternative embodiment 300, the microphone 310, the foot-input 

console 320, the control console 330, the system control and voice recognition 
processor 340, and the medical imaging acquisition and display system 350, 
function in substantially the same manner as described above with regard to Figure 
1. However, the enhanced display 360 of the alternative embodiment 300 includes 
extra functionality not present in the medical imaging acquisition and display 
system 1 50 of Figure 1 . The enhanced display 360 is preferably a LCD screen or 
monitor placed in clear view of the surgeon during surgery. In addition to 
displaying the medical images to the surgeon during surgery, the enhanced display 
360 may also display additional information to the surgeon. For example, the 
additional information displayed to the surgeon may include which functionality is 
currently assigned to each foot-input device, or the current state of the medical 
imaging control system during assignment of functionality to the foot-input 
device. The additional information may be displayed using in the form of text, 
icons, or preferably drop-down menus, for example. 

In operation of the alternative embodiment 300, the surgeon speaks the first 
verbal command into the microphone 31 0 as similarly described above with 
reference to Figure 1 . However, after the first verbal command is received and 
recognized by the system control and voice recognition processor 340, a drop- 
down menu preferably appears on the enhanced display 360. Once the 
functionality has been selected, the drop-down menu may either appear over the 
medial image being displayed or next to the medical image on the enhanced 
display 360. The top of the drop-down menu then preferably lists the name of 
functionality specified in the first verbal command. Below the name of the selected 
functionality listed in the drop-down menu is preferably the list of names of the 
assignable foot-input devices available for assignment on the foot-input console 
320. Thus, the surgeon is provided with a visual confirmation that the first verbal 
command was successfully recognized by the system control processor and voice 
recognition processor 340 when the drop-down menu appears on the enhanced 
display. Furthermore, the surgeon is then presented with a list of available foot- 



[0032] 
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input devices to choose from. 

[0033] Next, the surgeon speaks the second verbal command into the microphone 
310 as similarly described with regard to Figure 1. The second verbal command 
then assigns the commanded foot-input device specified in the second verbal 
command to the specified functionality of the first verbal command in substantially 
the same manner as described above in Figure 1 . Additionally, after the foot-input 
device is assigned functionality, the functionality assigned to the foot-input device 
is preferably continually displayed on the enhanced display 360. Displaying the 
assignment information on the enhanced display 360 allows the surgeon to review 
which foot-input devices have been assigned functionality, thus making the use of 
the medical imaging control system easier and more efficient. 

[0034] In a second alternative embodiment of the present invention, the enhanced 

display 360 described above in Figure 3 may be a head-mounted display instead of 
a LCD panel or monitor. A head-mounted display is typically a device worn on the 
operator's head which projects an image viewable by the operator. The image 
projected by the head-mounted display may either be projected directly into the 
operator's eye or reflected into the operator's eye using a mirror. The head- 
mounted display may be either a monocular head-mounted display, or a see- 
through head-mounted display. In either case the drop-down menu alone or 
preferably the medical image and drop-down menu may be displayed to the 
surgeon via the head-mounted display. The head-mounted display may allow the 
surgeon to have the patient, the medical images, and the medical imaging system 
information, in the surgeon's field-of-view during the surgery. Having all of this 
visual information in front of the surgeon may reduce the amount of head 
movements required by the surgeon to access all of this information. Reducing the 
number of head movements may increase the ease, speed, and efficiency of the 
surgery. 

[0035] 

In a third alternative embodiment of the present invention, the order of verbal 
commands may be reversed. That is, the first verbal command spoken by the 
surgeon selects the foot-input device to be assigned a function. Consequently, the 
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second verbal command spoken by the operator then selects the functionality of 
the medical imaging system to be assigned to the foot-input device in the first 
verbal command. The rest of the medical imaging control system in the third 
alternative embodiment functions in substantially the same manner as discussed 
above with regard to Figure 1 . 

[0036] While the invention has been described with reference to a preferred 

embodiment, it will be understood by those skilled in the art that various changes 
may be made and equivalents may be substituted without departing from the 
scope of the invention. In addition, many modifications may be made to adapt a 
particular situation or material to the teachings of the invention without departing 
from its scope. Therefore, it is intended that the invention not be limited to the 
particular embodiment disclosed, but that the invention will include all 
embodiments falling within the scope of the appended claims. 
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